Monnica Chan 
Tim O’Connor 
Stafford Peat 


September 2016 
New England 
Board of Higher 


Education 


www.nebhe.org 


Using Khan Academy in Community College 
Developmental Math Courses 


Results and Lessons Learned from 
Developmental Math Demonstration Project 


According to the U.S. Department of Education, 68% of community college students and 40% of 
students at four-year public institutions were required to take one or more remedial or developmental 
education courses before enrolling in college-level courses. The vast majority of developmental course- 
work that is taken is in math. According to a 2016 study by Education Reform Now,' the annual cost 
of providing remediation to all college students nationwide has been estimated at approximately $1.5 
billion. This figure accounts only for the direct costs of remediation and not students’ lost opportunity 
costs, including delayed completion or, more often, not earning a degree or credential at all. 


Using data from 57 community colleges participating in Achieving the Dream, a community college re- 
form network, the Community College Research Center at Columbia University followed the progress 
of 63,650 students through a developmental math course sequence. Results were dire: 


Only 26% of students who took a developmental course one level below college math 


successfully completed a subsequent college-level math course. 


A disturbingly low 11% of students who had to take a sequence of three or more developmental math 
courses eventually earned college-level math credit. 


About Developmental Math Demonstration Project 


The New England Board of Higher Education (NEBHE), with support from Lumina Foundation, 
investigated whether the use of Khan Academy could increase community college student success 
in developmental math coursework, support embedded math content in technical courses, and pre- 
pare students to take or retake college placement tests. Khan Academy, an Open Educational Resource 
(OER), is a nonprofit organization that provides free online educational resources such as practice 
exercises, instructional videos and a personalized learning dashboard that empowers learners to study at 
their own pace in and out of the classroom. 


Developmental Math Demonstration Project (DMDP) began in 2013 with the selection of 12 commu- 
nity colleges in five New England states (“pilots”). Participating institutions submitted a letter of intent 
to be part of the project and received small grants to offset costs associated with the project, including 
reporting to NEBHE on students using Khan Academy. 


1 Out of Pocket: The High Cost of Inadequate High Schools and High School Student Achievement on College Affordability, 
Education Reform Now. March 2016. 

2 Note: Analysis tracked 63,650 first-time, credential-seeking students for three years at 35 Achieving the Dream community 
colleges who began their enrollment from fall 2006 to fall 2008 and were referred to at least three levels of developmental 
education. Source: Shanna Smith Jaggars and Georgia West Stacey, “What We Know About Developmental Education 
Outcomes,” Community College Research Center, Teachers College, Columbia University, (Outcomes Research Overview/ 
January 2014). 


Pilot community colleges used Khan Academy’s math content as a supplement in various developmental education delivery models, 
including flipped, blended, self-paced, modular and traditional developmental education classrooms, as well as in some career and 
technical courses. A number of community colleges also used Khan Academy in Accuplacer math “boot camps.” In New England, 
nearly all community colleges and four-year public institutions use the College Board’s Accuplacer exam as a placement test. Boot 
camps were run as short-term (usually three- to six-week) courses to help students prepare for Accuplacer or, in the optimal situa- 
tion, test out of developmental math. 


NEBHE supported community college instructors by developing DMDP resources on the NEBHE website. The DMDP resourc- 
es included five training webinars and multiple curriculum maps aligning Accuplacer and developmental math course topics with 
Khan Academy practice exercises and videos. NEBHE also deployed an implementation coach, who provided live and web-based 
introductory and advanced trainings to participating instructors. 


This brief reports on the effectiveness of the pilot efforts, student and faculty perceptions regarding using Khan Academy, 
and the challenges encountered during the project. Several types of data were collected during the project, including student and 
faculty surveys, aggregated student performance data, and input from faculty during training sessions and network meetings. 


About Khan Academy 

Khan Academy was created in 2006 as an exclusively online nonprofit educational Khan Academy Core Values 
organization whose mission is “to provide a free, world-class education for anyone, Learning should be: 
anywhere.” The Khan Academy website welcome screen asserts, “You only have to know 
one thing: You can learn anything.” 


Personalized 


Mastery-based 


Interactive and exploratory 


The major components of the Khan Academy platform include: eden 


e <A personalized learning engine to help students keep track of what they are 
learning and to inform instructors (“coaches”) of their students’ learning gaps. 
Instructors can guide their students using Coach Recommendations.’ 


e An extensive library of content videos to help students review, move ahead and take ownership of their own learning. 


e Automated Practice Exercises to help students “level up” through their learning as they work toward mastery of 
concepts. 


e A suite of student and instructor tools including: 
° Targeted content Missions* 
. Mastery Challenges” 


* Real-time Student Progress Reports® 


Khan Academy provides practice exercises, instructional videos, and a personalized learning dashboard in a number of content 
areas including science, computer programming, history, art history, economics, and math. Khan Academy materials are licensed 
under a Creative Commons copyright arrangement and all materials are available free at www.khanacademy.org. 


3 Coach Recommendations allow a Coach to recommend a skill from any math mission to a student or group. Coach Recommendations also help to inform 
the Mastery Challenges based upon the progress of an individual, group of students or the entire class. 

4 Missions guide students through specific content. In Missions, students learn at their own pace and use hints and videos to master skills. 

5 Mastery Challenges help students learn beyond term-term recognition toward deeper understanding and long-term retention. 

6 Student Progress Reports provide an instructor with a summary of the class as well as well as the progress of each individual student. Student Progress Re- 
ports are useful for diagnosing learning gaps and providing targeted interventions. 


Project Findings 


How Khan Academy was Used All institutions and faculty mem- 


bers that participated in the project 


From summer of 2013 through spring 2015, 1,226 students and 37 faculty members tailored Khan Academy to meet their 
participated in the project. Approximately one-third of all DMDP students partic- needs. 
ipated in the fall 2013 semester. Institutional participation in the project was not 


evenly distributed over the course of the project due to two primary reasons: 


1. Some institutions made changes to their developmental math programs, or dropped out of DMDP because of shifting admin- 
istration priorities. 

2. Khan Academy made significant changes during and after the fall 2013 semester. A number of faculty members commented in 
faculty surveys and project meetings that the changes were disruptive to their class planning and course delivery and negatively 
affected student learning. As a result, they stopped using Khan Academy. 


Institutions and faculty members that participated in the project were allowed to tailor Khan Academy to meet their needs. This 
included using Khan Academy as a tool for placement test preparation, as a supplemental resource in college-level classes, and as a 
primary instructional resource in developmental math courses. Faculty members further personalized their use of Khan Academy 
by selectively using Khan Academy materials and tools. For example, some faculty members assigned only videos to students. Oth- 
ers utilized the entire platform, including its user tools and content scaffolding through Khan Academy “Missions.” 


User Experience 


Faculty feedback in project surveys and meetings highlighted the promise of Khan Academy. While several faculty members point- 
ed to small errors (e.g. a mistake in a video or an odd order of scaffolding material), faculty were generally enthusiastic about using 
Khan Academy in their classrooms because of the “replay” feature of the videos and “gigantic cache” of exercise problems. Even so, 
Khan Academy was not the most effective resource for every instructor. Some “missed the human component of the student-teach- 
er relationship” and found it difficult to entice students to use the tool. Others found that Khan Academy enhanced their relation- 
ships with students because it helped them “assess the strengths and weaknesses on a more individualized level with the students in 
the course.” 


Students also had mixed, but generally positive, experiences with Khan Academy. On student feedback surveys from academic 
year 2014-15, the most recent academic year and last year of the funded pilots, 77% of students agreed or strongly agreed with the 
statement, “using Khan Academy has helped me feel more confident when it comes to understanding and doing math.” A majority 
of student respondents also agreed with statements such as “using Khan Academy made this math class more enjoyable than others 
I’ve taken” (68%) and “I like using Khan Academy more than using a textbook” (73%). Students appreciated “being able to review, 
fast forward, rewind, go at [their] own pace” and work on practice problems using Khan Academy videos and exercises. While 
promising, Khan Academy wasn’t the best instructional tool for every student. In fact, a majority of students (67%) agreed with 
the statement, “Khan Academy can’t replace my professor.” Over a third of student survey respondents (36%) agreed that, “Khan 
Academy can’t replace a textbook.” Some students found the videos were “too fast,’ “confusing” or duplicative of other material 
already being used in class. Others “don’t like online classes” and prefer “a book and teachers teaching the book.” 


User Results 


Participating colleges reported course grades and Khan Academy usage data for 945 students. Course grading and Khan Academy 
usage varied across institutions, making it difficult to determine the effect of Khan Academy usage on students’ progression in and 
through developmental math curricula. There was, however, a statistically significant positive correlation between students’ activity 
on Khan Academy, particularly the number of mastery exercises completed, and their course grade (Figure 1). 


This confirms reports from some faculty members that Khan Academy was assigned for a grade in their classrooms. It may also 
suggest that Khan Academy did help students increase their understanding of course materials. 
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Figure 1: Sample Correlations between Selected Student Outcomes and Khan Academy Usage 


A B Cc D E F G H 
Accuplacer Number of 
Arithmetic Accuplacer Number of Vid- Time Spent on Time Spenton Exercises 
Variable Course Grade Gains Algebra Gains eos Watched Videos Exercises Mastered 


Course Grade - 


Accuplacer Arithmetic Gains - 


Accuplacer Alegbra Gains 0.183 - 

Number of Videos Watched 0.067 -0.300 0.211 - 

Time Spent on Videos 0.0890* -0.127 0.054 0.9381*** - 

Time Spent on Exercises 0.1763** 0.060 0.2718** 0.5079*** 0.4600*** - 

Number of Exercises Mastered 0.1916** -0.120 0.3042** 0.2593*** 0.1876*** 0.6226*** - 


*p<0.05; **p<0.01; ***p<0.001 

Note: Correlations do not include courses graded on a pass/fail basis. Accuplacer gains were calculated by subtracting post-course scores from pre-course scores provided by each institution. Courses with 
Accuplacer pre- and post-scores were non-graded courses. 

Source: NEBHE analysis of data from DMDP institutions 


Nearly 47% (351 of 749) of students took a math course the semester following their DMDP math course (in which they used 
Khan Academy). Of these students, 64% enrolled in a developmental math course. 


Khan Academy’s role in supporting students’ mastery of mathematical concepts is perhaps most explicit in the positive association 

between students’ activity on Khan Academy and gains in Accuplacer Algebra test scores (Figure 1, column D). It is unclear if these 
same students would have had similar gains under less individualized instruction as 
provided by participating faculty members or if another resource, other than Khan 


Academy, was used. Based on these results, however, supporting students to study Supporting students to study for 

for placement exams using Khan Academy videos and exercises may be one way to placement exams using Khan Acad- 
reduce remediation rates or help reduce the number of remedial courses a student emy videos and exercises may be one 
would need to take. way to reduce remediation rates or 


help reduce the number of remedial 


courses a student would need to take. 


Boot Camp Math Courses 
Five DMDP partner institutions supported students studying for placement exams using Khan Academy videos and exercises 


through semester- or weeks-long “boot camps.” NEBHE'’s analysis of data from these institutions found a moderate correlation 
(0.3010, p<0.01) between students’ Accuplacer Algebra gains and their enrollment in another math course (Figure 2, cell C4). 


Figure 2: Sample Correlations between Selected Student Outcomes and Khan Academy Usage in Boot Camp Courses 


A B C D E F G H 
Accuplacer Enrolled in Number of Number 
Arithmetic Accuplacer Passing the Another Math Time Spenton Exercises Mas- of Videos 
Variable Gains Algebra Gains Course Class Exercises tered Watched 


Accuplacer Arithmetic Gains - 


Accuplacer Alegbra Gains 0.183 - 

Passing the Course 0.083 0.147 - 

Enrolled in Another Math Class 0.144 0.3010** 0.3175** - 

Time Spent on Exercises -0.060 0.2718** 0.5026*** 0.1918* = 

Number of Exercises Mastered -0.120 0.3042** 0.168 0.045 0.5392*** - 

Number of Videos Watched -0.300 0.211 0.231 0.6652*** 0.4384*** - 


*p<0.05; **p<0.01; ***p<0.001 
Note: Boot camp courses were graded on a pass/fail basis. Accpulacer gains were calculcated by subtracting post-course scores from pre-course scores provided by each institution. 
Source: NEBHE analysis of data from DMDP institutions 


Project Findings — Key Takeaways 


1. Faculty participating in DMDP were given leeway on how Khan Academy was to be 
used in their class. As a result, a number of instructional approaches were employed. 


Faculty feedback was generally positive. Many responded that they liked the manage- 
ment features of Khan Academy. 


Students liked many of its features, including video playbacks. However, a significant 
number reported that Khan Academy couldn’t replace an instructor or textbook. 


Because participating faculty use of Khan Academy varied along with different grading 
processes it was difficult to judge the impact. However, there appeared to be a general- 
izable association with students’ successfully completing mastery exercises and course 


grades. 


Case Studies 


The following case studies highlight a few of the ways Khan Academy can be used in developmental math classrooms. Each case, in 
its unique way, exemplifies how different instructors achieved an effective implementation. 


Community College of Vermont 
Primary Resource Delivery Model 


Over the course of DMDP, two Community College of Vermont (CCV) instructors evolved from using Khan Academy as a 
supplement to the textbook to using Khan Academy as their primary instructional resource. During the last two semesters of the 
project, CCV combined a sequence of a semester of Foundations of College Math and a semester of Foundations of Algebra into 
one condensed course meeting six hours per week. This allowed students to use the Khan Academy Pre-Algebra and the Algebra 
Basics Missions back-to-back and complete two developmental math courses in one semester. 


CCV Outcomes 
One of the primary positive outcomes from CCV’s implementation is that instructors began to understand and incorporate The 
Four Essentials of an Effective Khan Academy Implementation (Sce page 8). Student evaluations suggested that instructor efforts 


to guide and motivate their students were, on the whole, successful. Instructors provided multiple levels of student support by: 


e Explaining and demonstrating the value of mastery learning 


¢ Students expressed that when they understood more about their CCV Outcomes 
own learning process, learning math became, in some cases, less intim- 
ae SP 8 Most students took ownership of their 
idating. 
own learning by completing Coach 
e Providing opportunities and activities to engage in peer collabora- Recommendations and working toward 
tion, productive struggle and growth mindset Mastery of topics. 


Most students consistently worked toward 
completing Pre-Algebra and Algebra Basics 


Missions. 


+ CCV uses Peer Mentors embedded in developmental math class- 
rooms. In addition to having Peer Mentors, in many classes students 
formed their own study groups. 


Many students independently formed their 


own cooperative study groups. 


Manchester Community College - Connecticut 
Blended Delivery Model 


In this model, Khan Academy was used to identify strengths and needs in students’ mathematical foundations. The instructor used 
Coach Recommendations, Mastery Challenges and Student Progress Reports to meet student learning needs. 


The content was delivered to small classes (<20 students) of Adults in Transition, a free course that aims to help adults succeed in 
college. Instruction was hybrid (blended), instructor-directed, and self-paced. 


Manchester CC Outcomes 
After three semesters of implementation, the instructor found consistent 


trends. Approximately 60% of students moved up at least one level on the 
Accuplacer exit exam. Of those students who placed into a higher math 


Manchester CC Outcomes 


level, two-thirds moved up two levels. When evaluating skills mastered on 
Khan Academy versus subsequent math placement, the instructor found 
that mastery of the coach recommended skills had a significant correlation 
with improved scores on Accuplacer, while general time spent on Khan 


Sixty percent of 119 participating students 
successfully placed into the next course in 


their math sequences. 


Of those students who placed higher via 


Academy without mastery did not necessarily correlate with improved Accuplacer scores, two-thirds moved up 


scores. two math levels. 


Mastery of coach recommended skills cor- 
related with higher Accuplacer scores. 


Nashua Community College - New Hampshire 
Boot Camp Delivery Model 


This model used a boot camp approach to prepare students to take or retake the Accuplacer and provide an opportunity to test out 
of developmental math coursework. 


Classes were free for students, and initially delivered in three-to-four week sessions. This approach evolved into three-hour self- 
paced yet proctored blocks for four days. 


At the beginning of DMDP, the instructor developed his own Khan Academy practice exercise playlist and video assignments. 
Midway through the project, Khan Academy introduced Missions, most notably the Algebra Basics Mission, which eliminated the 
need for each instructor to develop his or her own playlist. 


Nashua CC Outcomes 


Over the course of DMDP, the instructor ran seven boot camps of ap- 
proximately nine students per session. More than half of all students tested 
out of at least one developmental math level. In each boot camp, at least 
one student tested into college-level math and a few students tested out of 
pre-algebra. Students with a strong existing foundation in math were more 
likely to be successful in navigating the condensed format. 


Nashua CC Outcomes 


At least half of all students tested out of 
at least one or more developmental math 
levels. 

At least one student per boot camp tested 
into college-level math. 

Students with a strong existing foundation 
in math were more likely to successfully 


navigate the condensed format. 


Challenges Encountered 


From a pedagogical perspective, technology is increasingly being used in higher education to supplement instruction and deliver 
content. Khan Academy is emblematic of this paradigm shift. 


Blending good teaching with Khan Academy’s learning tools and assessment algorithms can be initially challenging for both stu- 
dents and instructors. Beyond becoming intimately familiar with the teaching and learning tools, using Khan Academy effectively 
requires students and instructors to recognize and embrace their new roles. For many instructors, engaging in the paradigm shift 
comes with a set of requirements that may require additional staff training. 


e Instructors should see themselves as coaches who guide, motivate and inspire their students. 


e Students should see themselves as active, self-directed learners who, with guidance, set their own learning goals and 
take responsibility for their education. 


e Students and instructors should also work together and, if possible, with peer mentors/tutors to share their learning 
and deepen their own content knowledge. 


In spring 2013, Khan Academy began a dramatic overhaul of its math platform. Many changes were made to respond to instructors’ 
requests to better manage and differentiate student learning. However, many of the changes were launched in beta and then pulled 
down mid-semester and revised, or, in some cases, no longer made available. Major changes included: 


e Revised Khan Academy Dashboard, including updated resources 
e Revised student and skills progress reports for individuals and groups of students 


e Launching of Missions, Mastery Challenges, and Coach Recommendations tools 


From spring 2013 to fall 2014, the constant platform revisions frustrated many of the math faculty and students and, in one case, 
was the precipitating factor in a community college dropping out of the project. Despite these challenges, faculty reported that by 
2015, the changes dramatically improved the functionality of the resource. 


Project Recommendations 
For Faculty — Use the Four Essentials of an Effective Khan Academy Implementation 


Developmental Math Demonstration Project findings can be valuable to all those 
seeking new pedagogies and tools to increase success in developmental math. Overly 
optimistic instructors may attempt to use Khan Academy as a silver bullet and run Start slowly and grow into the Khan 
the risk of underutilizing the tool’s collaborative capabilities, thereby leaving their Academy system. 

students to be entirely self-directed. Overly pessimistic instructors may feel chal- 
lenged by the initial learning curve in using Khan Academy or feel that the system is 
meant to replace them as a teacher. In either case, both instructors and students may miss out on the intended value of using Khan 
Academy. 


Cautiously optimistic instructors may question Khan Academy’s value for their students, but in the process of using the tool may 
see the value of using mastery learning, real-time student progress reports and the coach recommendations tool to individualize 
their instruction. 


Over time, trained instructors can improve student outcomes by using the Four Essentials of a Successful Khan Academy Imple- 
mentation. 


The Four Essentials of a Successful Khan Academy Implementation: 


Use the Coach Recommendation tool to guide students and hold them accountable 


Grade on Mastery of topics or targeted Missions 
Teach students the value of mastery learning and how to use Khan Academy effectively 


Give students support with peer tutors and growth mindset activities 


For Institutional Leaders 


Institutional leaders play a critical role in encouraging or discouraging instructional innovation. Ideally these innovations should be 
evidence- or research-based, but using this strategy alone will leave out extraordinary innovations and tools being developed based 
upon technological advancement. One of the best innovation examples is the burgeoning development and use of Open Educa- 
tion Resources like Khan Academy to drive down student textbook costs. Using this approach often requires additional staff time, 
revisions in instructional resources, and patience to find the best teaching and learning approaches. 


To support faculty in effectively using these resources, instructional leaders can: 


e Support and incentivize faculty to use the most effective instructional tools and pedagogical approaches 


e Provide time for faculty to share what works and what doesn’t work 
in various contexts, within and across campuses 


e Help faculty create investigation teams to explore new learning inno- Instructor success with Khan Academy is de- 
vations or approaches pendent upon knowing how to effectively use 


the tools to personalize instruction. 


e Use department meetings for sharing best practices 


Lessons Learned 


Support from college administrators matters—a great deal. Faculty members need more than one semester to find and develop 
their most effective use of Khan Academy. Faculty prep time increases during the early implementation period and decreases dra- 
matically by the third semester. 


Khan Academy is not a “silver bullet.” It may or may not adequately replace a textbook. It cannot substitute for a good instructor 
and it might not be for every faculty member or student, but it does address multiple learning styles and reinforces good teaching 
and learning habits. The immediate feedback, granular real-time progress reports, targeted learning missions, coach-guided learning 
recommendations, and emphasis on mastery learning are motivational for both students and instructors. 


Used effectively, Khan Academy can help developmental math students meet success. 


For questions or comments, please contact Stafford Peat at speat@nebhe.org or 617-533-9509. 


About the New England Board of Higher Education (NEBHE) 

Established in 1955 by six visionary New England governors, NEBHE is a regional compact that works across New England to: help leaders assess, 

develop and implement education practices and policies of regional significance; promote regional cooperation that encourages efficient sharing of 

education resources; and strengthen the relationship between higher education and the regional economy. 

Learn more at www.nebhe.org. @nebhe 


